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This paper describes how Appreciative Inquiry was used to design an organization ’s evaluation 
system that would contribute to the staff’s capacity to engage in evaluation and to think 
evaluatively. By building on participants’ peak evaluation experiences, their values and wishes 
with regard to evaluation, and what is commonly accepted as “good” evaluation practice, a 
comprehensive evaluation system was developed. The data collection process, as well as each of 
the system ’s components and elements is described. 
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The field and profession of evaluation is experiencing unprecedented growth. In the year 2000, the American 
Evaluation Association, ( www.eval.org ) had 2,463 members. Today, it is a vibrant and thriving organization of 
nearly 5,000 members who represent 67 countries. The Association’s annual conference held every November now 
draws over 2500 participants from over 40 countries - academics and practitioners who come together to share their 
evaluation experiences and to learn new theories and strategies. Evaluation is also growing around the world; in 
1980, there were three national and regional evaluation societies. In 1990, there were five, in 2000 there were 50, 
and today, there are more than 70 professional evaluation organizations. Unfortunately, however, the HRD field has 
been painfully slow to recognize that there is such a rich and relevant set of evaluation theories, practices, and 
ethical standards, beyond Kirkpatrick’s four-level taxonomy (1994). Yet, the pressure for learning and organization 
development professionals to develop evaluation competencies (Stevahn, King, Ghere, & Minnema, 2005) is 
mounting (Bassi & Lewis, 1999; Bober & Bartlett, 2004; Brown & Seidner, 1998; Russ-Eft & Preskill, 2001). 

An emerging trend in the evaluation field is a focus on building the evaluation capacity of organization 
members. Stockdill, Baizerman, and Compton (2002) describe evaluation capacity building as, “A context- 
dependent, intentional action system of guided processes and practices for bringing about and sustaining a state of 
affairs in which quality program evaluation and its appropriate uses are ordinary and ongoing practices within and/or 
between one or more organizations/programs/sites” (p. 8). Gibbs, Napp, Jolly, Westover, and Uhl (2002) further 
suggest that evaluation capacity building is, “The extent to which an organization has the necessary resources and 
motivation to conduct, analyze, and use evaluations” (p. 261). Consequently, the evaluator’s role in building 
evaluation capacity is to support members’ efforts to: 

• Conceptualize, design, and sustain ongoing evaluations. 

• Link evaluation to organization’s strategic mission and goals. 

• Connect evaluation to other forms of organizational inquiry. 

• Develop evaluation knowledge and skills. 

• Integrate evaluation practice within the organization’s systems, structures, and culture. 

• Learn at the individual, team, & organizational levels. 

• Use evaluation findings. 

• Think evaluatively. 

Building evaluation capacity may have a variety of benefits. For example, organization members may be better 
positioned to show results for accountability purposes as well as for seeking additional funding, they may be better 
able to align their efforts with other organizational process improvement initiatives, and their engagement in 
systematic and ongoing evaluation activities may help them make more timely and effective decisions. Ultimately, 
building organization members’ evaluation capacity can facilitate individual, team, and organizational learning 
(Preskill & Torres, 1999). 

Concurrent with the field’s commitment to building evaluation capacity, and perhaps as an outgrowth of 
evaluators’ preferences for participatory, collaborative, and democratic forms of evaluation, there is a growing 
interest in understanding what and how stakeholders learn from their involvement in the evaluation process (Forss, 
Rebien, & Carlsson, 2002; Patton, 1997; Preskill & Torres, 1999; Preskill, Zuckerman & Mathews, 
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(2003). Patton (1997) has described this kind of learning as process use. Process use occurs as individuals work 
together on an evaluation and participate in its design and implementation, and experience the issues and challenges 
that are inherent in evaluation practice. In the end, it can be said that participants (stakeholders) learn as much about 
evaluation practice as they do about the findings of the evaluation. Preskill and Torres (1999) suggest that evaluation 
is a process for enhancing knowledge and decision-making within organizations and communities. It involves 
answering questions and/or addressing issues through the collection and analysis of data about programs, systems, 
processes, procedures, products, and services; that evaluation is best implemented as a systematic process that is 
planned and purposeful, and with a clear intention of using the evaluation findings. Finally, evaluation is a means 
for understanding what we do and the effects of our actions in the context of the work environment and the society 
in which we live. This view of evaluation emphasizes that evaluation can produce much more than a report of 
findings, and rather than being event-driven, evaluation should be ongoing and part of everyone’s job - that all 
organization members have the ability to think evaluatively. As a result, evaluative inquiry for learning focuses on: 

• Program and organizational processes as well as outcomes; 

• Shared individual, team and organizational learning; 

• Education and training of organizational practitioners in inquiry skills; 

• Collaboration, cooperation, and participation; 

• Establishing linkages between learning and performance; 

• Searching for ways to create greater understanding of the variables that affect organizational success and 
failure; and, 

• Using a diversity of perspectives to develop understanding about organizational issues. (Preskill, 2005, p. 
146) 

Viewing evaluation as a catalyst for learning, and acknowledging that an increasing number of learning and 
organization development professionals are being asked to conduct evaluations of their own programs, products, 
processes, and services, it seems more important than ever, that we begin to think about how to build evaluation 
systems that maximize individual, group, and organizational learning. 

Evaluation Design and Methods - Building and Sustaining an Evaiuation System 

Participants and Setting 

Young Audiences of Northeast Ohio (YANEO, formerly Young Audiences of Greater Cleveland) is an affiliated 
professional chapter of Young Audiences, Inc., the country’s oldest and largest nonprofit arts-in-education 
organization dedicated to making the arts essential to the education of all school-aged children. In 2005-2006, 
through more than 6,100 programs, YANEO served over 265,000 students in urban, rural and suburban school 
districts throughout six Northeast Ohio counties ( http ://www .vaneo.org/who/who .html ) . The organization provides 
demonstrations and performances that introduce students to an art form and support various curricular areas. 
Ultimately, the hope is that students get excited about art. In an effort to continuously monitor and improve the 
processes, programs, activities, outcomes, and impact of their programs and services, the organization’s Executive 
Director commissioned the development of an evaluation system. Her ultimate goal is to build the internal 
evaluation capacity of the organization’s members so that they will have the knowledge and skills to conduct timely 
and useful evaluation studies. 

Using Appreciative Inquiry to Design The Evaluation System Design 

The content of the evaluation system was derived from three sources: 1) what is considered effective and ethical 
evaluation practice by professional evaluators, 2) an earlier systems model developed by Preskill and Russ-Eft 
(2003), and 3) information gained from YANEO staff and others who attended a one-day evaluation planning 
meeting at the YANEO office. The purpose of the meeting was to gather information from various stakeholders 
about what an effective evaluation system would look like. Given the Executive Director’s interest in building the 
capacity of the staff and other stakeholders to conduct evaluations and to think evaluatively, it was agreed that the 
most cost effective and potentially valuable approach would be to use Appreciative Inquiry (Al). Appreciative 
Inquiry is a process that inquires into, identifies, and further develops the best of “what is’’ in organizations and 
communities in order to create a desired future (Cooperrider, Whitney, & Stavros, 2003). While Appreciative 
Inquiry has been used in numerous and widely varied contexts around the world, only recently has it begun to 
appear in the evaluation literature as an approach for designing and conducting evaluations (Preskill & Catsambas, 
2006; Preskill & Coghlan, 2003). We believed that the use of Al would not only offer participants a respectful and 
safe way to raise issues and concerns about current evaluation practices, but it would also help them envision an 
evaluation system based on their previous positive/successful evaluation experiences. With input from the facilitator 



(this paper’s author) regarding the size and makeup of the stakeholder group, the Executive Director invited ten staff 
members, two board members, and one artist, to participate. During the one-day meeting they were guided through 
each of the four AI phases with a series of activities and questions. Participants were seated at two tables (one with 
six participants, the other with seven). 

Results 

During the first phase of the AI process. Inquire, participants were asked to consider the following questions and to 
interview each other in pairs. 

Peak Experiences'. Think of a time when you knew that an evaluation process was working well. You were 
confident and excited that important and useful data were being collected and you felt energized about what you 
were learning. What was happening? Who did it involve? What made this evaluation process (or outcome) so 
successful? What was your role? What was the core factor that made this evaluation process so effective? 
Values'. What do you value most about evaluation? What do you value most about yourself with regard to 
collecting and using information? 

Wishes: If you had 3 wishes for having more peak experiences with evaluation, like the one you just described, 
what would you wish for? 

Participants then told the highlights of their partner’s story to the others at their tables. From their stories, they 
identified themes that emerged from their peak experiences, values, and wishes, and listed those on flipchart paper. 

In the second phase. Imagine, participants were asked to consider the following, first individually, and then with 
others at their table: 

Imagine that you have been asleep for three years, and when you awake, you look around and see that the 
organization has developed a comprehensive, effective, and efficient evaluation system. This system provides 
timely and useful information for decision-making and action relative to the organization’s programs and 
services. The evaluation system has been so successful that the Arts Education Partnership has decided to 
bestow on Young Audiences of Cleveland its first ever, outstanding evaluation practice award. After hearing 
about your award. The Today Show invited your team onto the show to discuss the organization’s evaluation 
system. 

Fast forward one month - you are in New York and have just arrived at the NBC studios (driven by stretch limo 
from your hotel), and it’s now time for lights, cameras, action! Ann Curry begins the interview by asking you to 
describe what the evaluation system does, how it works, the kinds of information it collects, who uses the 
information, and how the information is used. What do you tell her (and the millions of viewers across the 
globe)? 

Again, the groups identified and wrote on flipchart paper, the themes that emerged from their visions. 

During the third phase. Innovate, participants were asked to individually develop provocative propositions - 
statements written in the present tense that reflect the organization’s social architecture (leadership, strategy, 
structures, systems, communication, management practices, internal customer relations, culture, people, values, 
competencies, roles and responsibilities, or business processes), that would support their visions of an evaluation 
system becoming a reality. Participants were told to write these statements in ways that stretch the status quo and 
are actionable. The statements, which were written on post-its notes were placed on flipchart paper around the 
room, and were then organized into seven themes and labeled. For the fourth and final phase. Implement, 
participants were asked to develop two or three recommendations for what would need to happen to make these 
provocative propositions become a reality. These were collected by the facilitator (this paper’s author) and were 
included as data in the evaluation report. 

The YANEO Evaluation System 

Figure 1 depicts the evaluation system that resulted from the data collected from the one-day meeting, and 
commonly agreed upon core elements and processes associated with professional evaluation practice. When 
designing and implementing an evaluation system, it is first important to recognize that whatever system any 
organization implements, it will always be situated within a larger organizational context that is, and will be, 
influenced and affected by external forces. For YANEO, it was determined that these forces may include 
expectations and requirements made by the Board of Directors, Funders and Donors, changing Legal Requirements, 
increasing Competition from other arts-based organizations, and fluctuations in the Workforce and Volunteer pools. 
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Establishing a Direction 

Next, every evaluation system should communicate an overarching vision and strategy for integrating evaluation 
into the organization’s work practices and other systems. 

Evaluation Vision Statement 

A vision statement for evaluation describes the values and beliefs the organization has for evaluative thinking and 
practice. It communicates the role evaluation plays in organizational decision-making, why evaluation is important, 
and how evaluation contributes to the future of the organization. The vision statement should be communicated in a 
variety of formats with all of those involved in the organization’s programs, activities and services, and should be 
the basis for all evaluation work. The vision statement should also be reviewed annually to determine if changes are 
necessary. 

Strategic Evaluation Plan 

A Strategic Evaluation Plan describes how, when, by whom, and to what extent various YANEO programs, 
services, processes, or policies will be evaluated. Decisions on when to evaluate may depend on various factors 
including: a) the length of time the program has been in operation, b) how the findings will be used - what kinds of 
decisions need to be made, and c) how often the program is offered. A Strategic Evaluation Plan helps the 
organization be proactive about its evaluation processes and resources. The plan’s development should involve 
YANEO staff in determining where the greatest evaluation needs are, when various programs should be evaluated, 
how often, and through which methods. 

Developing the Organization ’s Infrastructure for Evaluative Inquiry 

In order for an Evaluation Vision and an Evaluation Strategic Plan to be implemented, there are at least four 
elements of an organization’s infrastructure that need to be developed and sustained. These include: a) committed 
Leadership, b) an Evaluation Culture, c) sufficient Resources, and d) user-friendly and effective Technologies. The 
system is predicated on a belief that if any of these elements are missing, then the evaluation system’s effectiveness 
may be significantly compromised. 

Leadership 

For an evaluation system to work - for evaluation to be integrated into how the organization accomplishes its goals, 
it is absolutely critical that the organization’s leaders truly believe that evaluation is a meaningful and important 
activity. As such, they must consistently: a) communicate the importance of evaluation for decision-making, b) 
encourage organization members to engage in asking questions, reflection, and dialogue, and c) provide the 
necessary time, personnel, and financial resources to ensure that evaluations can be conducted. Leaders who support 
evaluation: 

Are champions of evaluation - they actively engage in, and encourage others to think evaluatively. 

Provide resources (financial, time, personnel) for conducting quality evaluations. 

Demonstrate the value of evaluation by using evaluation findings to make decisions. 

Use findings to make improvements to the organization’s programs, processes, policies, and systems. 

Use findings to share lessons learned and to determine next steps. 

Celebrate and publicize how an evaluation’s findings are being used to further the organization’s goals and 
overall health. 

Communicate to both internal and external audiences that evaluation is a priority. 

Clearly articulate the staffs roles and responsibilities regarding how, when, and by whom, evaluations are 
to be conducted. 

Provide reward and recognitions systems that value staffs engagement in evaluation-related activities. 

To support evaluative inquiry and practice, leaders may develop a business plan for obtaining the necessary 
resources for implementing the evaluation system, communicate internally and externally, how the evaluation 
system was developed, communicate the benefits of integrating evaluation into the work of organization members, 
show how the evaluation system fits in with other organization-wide development and improvement efforts, and 
ensure that the organization’s reward, recognition, and compensation systems reflect the organization’s commitment 
to the staffs engagement in evaluation work. 

Evaluation Culture 

For an evaluation system to be integrated into an organization’s work, it is imperative that a culture of evaluation be 
established, nourished, and sustained. Specifically, an evaluation culture supports: a) employees’ ability and 
willingness to take risks, trust and respect one another, give and receive feedback, b) the ongoing use of evaluation 
findings for decision making and action, and c) the learning that may occur from stakeholders’ involvement in the 
evaluation process. 




Developing an evaluation culture is not something that is accomplished over night, nor does it occur within a 
vacuum. An evaluation culture reflects the attitudes, skills, and interests of all organization members (and those 
involved in conducting evaluations for the organization). While the organization’s leadership is instrumental in 
modeling the behaviors associated with a culture of inquiry, staff members must understand their role in creating 
and maintaining the desired evaluative culture. 

Personnel and Financial Resources 

Evaluation systems will only succeed in serving the organization if there are adequate personnel and financial 
resources dedicated to the evaluation effort. In terms of personnel, this translates into having enough people to carry 
out the evaluations, as well as the ability to hire outside consultants to train and facilitate certain tasks of the 
evaluation if necessary. In addition, if the goal is to build organization members’ capacity to think evaluatively and 
to conduct evaluations, then the staff should be encouraged to participate in evaluation related professional 
development opportunities. An evaluation system also requires the allocation of financial resources so that staff 
may conduct quality and timely evaluations. These resources may result in hiring more internal staff, reallocating 
work loads, purchasing software and other technologies, travel, hiring external consultants, and other associated 
administrative costs. 

Technology 

Technology plays an important role in collecting and analyzing evaluation data. Technology is also instrumental for 
disseminating information about an evaluation’s progress as well as its findings, recommendations, and action plans. 
Consideration should be given to the following factors when determining which technologies and software programs 
may be used for evaluation purposes. Technology that supports an evaluation system: 

• Reflects a user-friendly approach for developing online surveys. 

• Provides user-friendly data analysis procedures with a variety of analysis and summary options (for both 
quantitative and qualitative data). 

• Has the ability to develop reports that have standard and customizable features. 

• Has the ability to provide evaluation summaries that are accessible to the organization’s staff. 

• Allow employees to retrieve data and reports from their computers. 

• Is linked to other data systems within the organization. 

Designing and Implementing an Evaluation Process 

For any evaluation to be successful, it is critically important that a well thought out and articulated plan should be 
developed. This plan ensures that key evaluation questions will be addressed, that critical stakeholders’ information 
needs will be represented, and that appropriate and useful methods will be used to collect information. 

Evaluation Plans 

For evaluations to provide the most useful and meaningful findings, it is important to take the time to carefully plan 
an evaluation. Developing an evaluation plan ensures that: 

Every evaluation effort will address important questions. 

The intended uses and users of the evaluation findings will be considered. 

The purpose of the evaluation will be clear. 

The data collection methods will obtain valid and useful information. 

The timeline and budget will be reasonable. 

The following are components of an evaluation plan that can be used to guide the development and implementation 
of any evaluation. 

1 . Focusing the Evaluation 

Evaluation Rationale - Describes why the evaluation is being conducted and the factors that have led to the 
need or interest in the evaluation. 

Purpose - In two-four sentences, describes the purpose of the evaluation and how the results will be used. 
Stakeholders - Describes the various individuals and groups who have a vested interest in the design, 
implementation and outcomes of the evaluation. In the case of YANEO, these stakeholders might include 
internal staff, board members, educators, artists, parents, volunteers, donors, legislators, and the public at 
large. 

Key Questions - Identifies the questions the evaluation will seek to answer. These questions form the 
boundaries and scope of the evaluation effort. They are generally open-ended, broad questions that are 
specifically related to the stakeholders’ information needs. 




Program Logic Model - Is usually a graphic representation of how the program is supposed to work. A 
logic model articulates the program’s theory of action and/or change by illustrating the program’s 
underlying assumptions, activities, resources, outputs, and short and long-term goals. 

2. Designing the Evaluation and Determining Data Collection Methods 

Evaluation Design - Describes the a) evaluation model or approach that will be used (e.g., organizational 
learning, empowerment, behavioral objectives, utilization-focused), b) the evaluation’s design (e.g., case 
study, time series, survey, randomized control group or quasi-experimental), c) methods of data collection 
including information on any sampling procedures, d) drafts of each data collection instrument (e.g., 
interview guides, surveys, tests, observation and document recording forms), and e) an explanation of how 
the data collection instruments will be pilot-tested to ensure the data’s validity. 

3. Analyzing Evaluation Data 

Data Analysis - Describes how the data will be entered into a database, analyzed, and stored. 

4. Managing the Evaluation 

Evaluation Management Plans - Includes one or more management plans (e.g., data collection plan, roles 
and responsibilities plan, timeline). 

Communicating and Reporting Plan - Identifies who will receive communications about the evaluation as it 
progresses as well as who will receive the evaluation results, and in what format they will be provided. For 
evaluation findings to be used for decision-making and action, a variety of communication strategies should 
be used. In choosing from among these strategies, it is useful to consider the information needs of the 
various stakeholders as well as the formats that will be most effective (Torres, Preskill, & Piontek, 2005). 
Evaluation Budget - Outlines a comprehensive budget for the evaluation. 

5. Using Evaluation Findings 

Action plans - Establishes the ways in which and by whom the evaluation recommendations will be 
implemented. 

Developing an evaluation plan that incorporates these elements increases the potential that the evaluation results will 
be meaningful and used for the organization’s decision-making purposes. 

Implications for Practice and Research 

The evaluation system described in this paper is offered as a means for thinking about what an evaluation system 
would look like, and how such a system can be developed using an Appreciative Inquiry approach. By engaging 
participants in a process of co-creating the system based on their peak experiences with evaluation, the proposed 
evaluation system more likely reflects the specific cultural and organizational context in which their work is carried 
out. Using AI was also able to reinforce the fact that while everyone has had different and varied experiences with 
evaluation, they nonetheless share a common vision for how evaluation could contribute to their organization’s 
health and success. 

Even though an organization’s adoption of an evaluation system symbolically communicates the organization’s 
support for evaluation, in the end, an evaluation system will only be as effective as its ability to serve the 
information needs of its stakeholders. As such, the system’s success will be largely dependent on the leadership’s 
willingness and ability to provide the necessary support to carry out the evaluation activities. This not only includes 
financial and personnel resources, but it also means making evaluation a priority and thus allowing and encouraging 
staff to design and conduct evaluations as needed. Finally, staff must be provided with opportunities to engage in 
various kinds of evaluation related professional development opportunities. Whether they attend workshops or take 
courses on evaluation, whether they are provided with technical assistance, or enter into a mentoring relationship 
with an evaluator, building the capacity of organization members to think evaluatively and to conduct evaluations is 
an ongoing process. We must begin to help organizations understand that evaluation is more than implementing a 
survey at the end of training; it is more than conducting a few interviews after a leadership development exercise; 
and it is more than thinking in terms of “levels.” Working with organizations to develop evaluation systems is one 
way to broaden HRD practitioners’ ways of knowing and thinking about evaluation. And, Appreciative Inquiry is a 
cost effective, productive, and energizing way to collect the information needed to build such systems. 

Research on evaluation within the HRD context is shockingly absent. Few HRD researchers have tapped into 
the evaluation profession and discipline to better understand the ways in which evaluation can contribute to HRD 
theory and practice. Why we have allowed Kirkpatrick’s simplistic and narrow evaluation approach to dominate 
training program evaluation for nearly half a century is an enigma we should all pause to consider. The study 
reported in this paper is one attempt to help the HRD field think more broadly and systematically about the role and 




practice of evaluation within organizations. The results of this study raise several research questions for HRD 
researchers. These include: 

• How might an HRD evaluation system be aligned with other forms of data collection within an 
organization (e.g., Six Sigma, Balanced Scorecards, dashboard indicators, Baldridge)? 

• How might Appreciative Inquiry be used for other kinds of HRD evaluation? In what contexts would 
using an AI approach be particularly effective? 

• How does Appreciative Inquiry facilitate individual, team, and organizational learning? 

• What are the long-term effects of using Appreciative Inquiry as an evaluation method? 

• What might other kinds of HRD evaluation systems look like? 

Answers to these and other evaluation research questions might help the HRD field have a stronger voice and 
presence in organizational decision making arenas; building members’ evaluation capacity is about ensuring that 
HRD contributes to the organization’s strategy for success. 
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